Method and apparatus for decoding an enhanced video stream

ABSTRACT

A method of decoding an enhanced video stream composed of base layer video access units and enhancement layer video access units, each access unit comprising a plurality of syntax structures, includes passing the syntax structures of the base layer access units to a base layer buffer, passing syntax structures of the enhancement layer access units to an enhancement layer buffer, outputting the syntax structures passed to the base layer buffer in a predetermined sequence, outputting the syntax structures passed to the enhancement layer buffer in a predetermined sequence, and recombining the sequences of syntax structures output by the base layer buffer and the enhancement layer buffer respectively to form a complete enhanced access unit, composed of base layer syntax structures and enhancement layer syntax structures in a predetermined sequence.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. application Ser. No.15/834,055 filed on Dec. 6, 2017, which is a continuation of U.S. Pat.No. 9,841,272 issued on Dec. 26, 2017, which is a continuation of U.S.Pat. No. 9,167,246 issued on Oct. 20, 2015, which is a continuation U.S.Pat. No. 8,369,415 filed Feb. 5, 2013, which claims priority under 35U.S.C. § 1 19 (e) from earlier filed United States ProvisionalApplication No. 61/034,370 filed Mar. 6, 2008, the entire disclosure ofwhich is hereby incorporated by reference herein for all purposes.

BACKGROUND OF THE INVENTION

This invention relates to a method and apparatus for decoding anenhanced video stream.

Referring to FIG. 1 of the drawings, a video encoder 10 receives rawvideo data, typically in the HD-SDI format defined in SMPTE 292 M, froma source such as a camera. The video encoder utilizes the HD-SDI data togenerate a video elementary stream and supplies the video elementarystream to a video packetizer 14, which produces a video packetizedelementary stream (PES) composed of variable length packets. Typically,each packet of the video PES contains one or more video frames.Similarly, an audio encoder (not shown) receives raw audio data from,for example, a microphone and supplies an audio elementary stream to anaudio packetizer, which creates an audio PES composed of variable lengthpackets.

The video and audio packetizers supply the video and audio PESs to atransport stream multiplexer 18, which assigns different respectiveprogram identifiers (PIDs) to the video PES and the audio PES andorganizes the variable-length packets of the video and audio PESs asfixed-length MPEG-2 transport stream (TS) packets each having a headerthat includes the PID of the PES and a payload containing the PES video(or audio) data.

The single program transport stream (SPTS) that is output by thetransport stream multiplexer may be supplied to a program multiplexer 22that combines the SPTS with other transport streams, conveying otherprograms, to produce a multi-program transport stream (MPTS). The MPTSis transmitted over a channel to a receiver at which a programdemultiplexer 26 separates a selected SPTS from the MPTS and supplies itto a transport stream demultiplexer 30. It will be appreciated by thoseskilled in the art that the SPTS that is output by the transport streammultiplexer may be transmitted directly to the transport streamdemultiplexer without first being combined with other transport streamsto create the MPTS but in either case the transport stream demultiplexerreceives the transport stream packets of the selected SPTS and separatesthem on the basis of PID, depacketizes the transport stream packets torecreate the PES packets, and directs the video PES to a so-called videosystem target decoder (T-STD) 34 and the audio PES to an audio T-STD 38.The subject matter of this application is concerned with decoding avideo bitstream and accordingly we will not discuss the audio decoderfurther.

The video T-STD 34 comprises a system target decoder buffer 40 and avideo decoder 42. The STD buffer 40 is functionally equivalent to atransport buffer Tb, a multiplexing buffer Mb, and an elementary streambuffer Eb. The transport buffer Tb receives the video PES at a variablebit rate and outputs the data at a constant bit rate to the multiplexingbuffer Mb, which depacketizes the video PES and supplies an encoded bitstream at a constant bit rate to the elementary stream buffer Eb. Theelementary stream buffer, which is sometimes referred to as the decoderbuffer or as the coded picture buffer (CPB), receives the CBR bitstreamand holds the bits for decoding a picture until they are all removedinstantaneously by the video decoder at the picture decode time.

It is important to proper operation of the decoder that the decoderbuffer should neither overflow, so that bits are lost and a picturecannot be decoded, or underflow, so that the decoder is starved of bitsand is unable to decode a picture at the proper time. The supply of bitsto the decoder buffer is controlled by a compressed data buffer (CDB) 46that receives the bitstream from the video encoder 10. The video encodersupplies bits to the CDB at a rate that depends on the fullness of theCDB. The CDB supplies bits to the video packetizer 14 at a constant rateand the multiplexing buffer supplies bits to the decoder buffer at thesame rate, and accordingly the fullness of the CDB mirrors the fullnessof the decoder buffer. By adjusting supply of bits to the CDB so as toprevent overflow/underflow of the CDB, we avoid underflow/overflow ofthe decoder buffer.

The video compression standard governing operation of the encoder mayspecify that the CDB should be no larger than the decoder buffer of ahypothetical reference decoder.

The MPEG-2 transport stream is widely used for delivery of encoded videoover an error prone channel. The MPEG-2 system layer also provides fortransmission of encoded video in the program stream (PS) in an errorfree environment. FIG. 1 illustrates transmission of the video PES as aprogram stream to a video P-STD 50 as an alternative to delivery as atransport stream to the video T-STD 34.

The bitstream produced by the video encoder 10 may comply with the videocompression standard that is specified in ISO/IEC 14496-10 (MPEG-4 part10) Advanced Video Coding (AVC), commonly referred to as H.264/AVC.H.264/AVC uses picture as a collective term for a frame or field.H.264/AVC defines an access unit as a set of network abstraction layer(NAL) units and specifies that the decoding of an access unit alwaysresults in a decoded picture. A NAL unit of an access unit produced byan AVC encoder may be a video coding layer (VCL) unit, which containspicture information, or a non-VCL unit, which contains otherinformation, such as closed captioning and timing.

Annex G of H.264/AVC prescribes an extension of H.264/AVC known asscalable video coding or SVC. SVC provides scalable enhancements to theAVC base layer, and the scalability includes spatial scalability,temporal scalability, SNR scalability and bit depth scalability. An SVCencoder is expected to create an H.264/AVC conformant base layer and toadd enhancement to that base layer in one or more enhancement layers.Each type of scalability that is employed in a particular implementationof SVC may utilize its own enhancement layer. For example, if the rawvideo data is in the format known as 1080 HD, composed of frames of1920×1088 pixels, the base layer may be conveyed by a sub-bitstreamcomposed of access units that can be decoded as pictures that are704×480 pixels whereas an enhancement layer may be conveyed by asub-bitstream that is composed of access units that enable a suitabledecoder to present pictures that are 1920×1088 pixels by combining thebase layer access units with the enhancement layer access units.

A decoder having the capability to decode both a base layer and one ormore enhancement layers is referred to herein as an SVC decoder whereasa decoder that cannot recognize an enhancement layer and is able todecode only the base layer access units, and therefore does not have SVCcapability, is referred to herein as an AVC decoder.

An access unit produced by an SVC encoder comprises not only the baselayer NAL units mentioned above, which may be conveniently referred toas AVC NAL units, but also SVC VCL NAL units and SVC non-VCL NAL units.FIG. 2 shows the sequence of AVC NAL units and SVC NAL units in an SVCaccess unit as prescribed by the SVC standard. In the event that theencoder produces, for example, two enhancement layers, the non-VCL NALunits for the two enhancement layers are in adjacent blocks of thesequence shown in FIG. 2, between the blocks containing the AVC non-VCLNAL units and the AVC VCL NAL units, and the SVC VCL NAL units for thetwo enhancement layers are in adjacent blocks of the sequence after theblock containing the AVC VCL NAL units.

An SVC decoder that extracts the base layer NAL units from the accessunit selects only the AVC non-VCL NAL units and the AVC VCL NAL units.

H.264/AVC specifies a five-bit parameter nal_unit_type, or NUT. UnderH.264/AVC, AVC NAL units all have NUT values in the range 1-13. SVC addsNUT values 14, 20 and 15. However, a NAL unit having NUT equal 14immediately preceding NAL units having NUT equal 5 or 1 signals baselayer slices, such that these NAL units, which are non-VCL NAL units,are compatible with AVC and can be decoded by an AVC decoder.

Referring to FIG. 3, an SVC encoder 10′ generates a unitary bitstreamthat conveys the base layer and, for example, two enhancement layersENH1 and ENH2. Depending on its capabilities, a decoder might expect toreceive, and decode, the base layer only, or the base layer andenhancement layer ENH1, or the base layer and both enhancement layerENH1 and enhancement layer ENH2. Under the MPEG-2 systems standard anduse case for SVC, the encoder may not provide three bitstreams,conveying respectively the base layer only, the base layer andenhancement layer ENH1, and the base layer and both enhancement layerENH1 and enhancement layer ENH2 and allow the decoder to selectwhichever bitstream it is able to decode. The encoder must provide thebase layer access units and parts of each enhancement layer in separatebitstreams. It would be possible in principle to comply with the MPEG-2systems standard by using a NAL separator 48 to separate the unitarybitstream into three sub-bitstreams based on the NUT values of the NALunits. One sub-bitstream would convey the base layer NAL units and theother two sub-bitstreams would convey the NAL units for the twoenhancement layers respectively. The three sub-bitstreams would pass torespective video packetizers (generally designated 14), which createrespective video PESs. The three video PESs would be supplied to atransport stream multiplexer 18 including a T-STD buffer equivalent tothe buffer that is included in an SVC T-STD, for the purpose ofmultiplexing together the outputs of the three packetizers. Themultiplexer 18 would assign different PIDs to the three PESs and outputsa transport stream conveying the three layers.

The video T-STD34 shown in FIG. 1 is unable to decode the bitstreamconveyed by the transport stream produced by the transport streammultiplexer 18′ shown in FIG. 3 because it provides no capability toreassemble the base layer and enhancement layer access units to producea complete SVC access unit that can be decoded by an SVC decoder.Neither the H.264/AVC standard nor the MPEG-2 systems standardprescribes how the base layer and enhancement layer access units shouldbe reassembled. Therefore, the architecture shown in FIG. 3 has hithertolacked practical application.

SUMMARY OF THE INVENTION

In accordance with a first aspect of the disclosed subject matter thereis provided a method of decoding an enhanced video stream composed ofbase layer video access units and enhancement layer video access units,each access unit comprising a plurality of syntax structures, saidmethod comprising passing the syntax structures of the base layer accessunits to a base layer buffer, passing syntax structures of theenhancement layer access units to an enhancement layer buffer,outputting the syntax structures passed to the base layer buffer in apredetermined sequence, outputting the syntax structures passed to theenhancement layer buffer in a predetermined sequence, and recombiningthe sequences of syntax structures output by the base layer buffer andthe enhancement layer buffer respectively to form a complete enhancedaccess unit, comprising base layer syntax structures and enhancementlayer syntax structures in a predetermined sequence.

In accordance with a second aspect of the disclosed subject matter thereis provided a method of creating an enhanced video signal comprisingreceiving a unitary bitstream composed of base layer access units andenhancement layer access units, separating a base layer program streamand an enhancement layer program stream from the unitary bitstream, andinserting a delimiting syntax structure into the enhancement layerprogram stream.

In accordance with a third aspect of the disclosed subject matter thereis provided a method of creating an enhanced video signal comprisingreceiving a unitary bitstream composed of base layer access units andenhancement layer access units, wherein each enhancement layer accessunit comprises video layer syntax structures and non-video layer syntaxstructures, separating a base layer program stream and an enhancementlayer program stream from the unitary bitstream, and including non-videolayer syntax structures of the enhancement layer access units in thebase layer program stream, whereby the enhanced video signal comprises abase layer component that includes non-video layer syntax structures ofthe enhancement layer access units and an enhancement layer componentthat includes video layer syntax structures of the enhancement layeraccess units.

In accordance with a fourth aspect of the disclosed subject matter thereis provided a decoding apparatus for decoding a base layer programstream conveying a succession of base layer access units of an enhancedvideo stream and at least one enhancement layer program stream conveyinga succession of enhancement layer access units of said enhanced videostream, each access unit comprising a plurality of syntax structures,the decoding apparatus comprising a base layer buffer connected toreceive the base layer program stream and to output syntax structures ofeach base layer access unit in a predetermined sequence, an enhancementlayer buffer connected to receive the enhancement layer program streamand to output syntax structures of each enhancement layer access unit ina predetermined sequence, a reassembly functional element connected toreceive the syntax structures output by the base layer buffer and theenhancement layer buffer respectively and to form a complete enhancedaccess unit, comprising base layer syntax structures and enhancementlayer syntax structures in a predetermined sequence.

In accordance with a fifth aspect of the disclosed subject matter thereis provided apparatus for creating an enhanced video signal, theapparatus having an input for receiving a unitary bitstream composed ofbase layer access units and enhancement layer access units andcomprising a separator for separating a base layer program stream and anenhancement layer program stream from the unitary bitstream, and aninserter for inserting a delimiting syntax structure into theenhancement layer program stream.

In accordance with a sixth aspect of the disclosed subject matter thereis provided apparatus for creating an enhanced video signal, theapparatus having an input for receiving a unitary bitstream composed ofbase layer access units and enhancement layer access units, wherein eachenhancement layer access unit comprises video layer syntax structuresand non-video layer syntax structures, and comprising a separator forseparating a base layer program stream and an enhancement layer programstream from the unitary bitstream and including non-video layer syntaxstructures of the enhancement layer access units in the base layerprogram stream, whereby the enhanced video signal comprises a base layercomponent that includes non-video layer syntax structures of theenhancement layer access units and an enhancement layer component thatincludes video layer syntax structures of the enhancement layer accessunits.

In accordance with a seventh aspect of the disclosed subject matterthere is provided a computer readable medium containing software that,when executed by a computer having an input for receiving an enhancedvideo stream that conveys base layer access units and enhancement layeraccess units, each access unit comprising a plurality of syntaxstructures, processes the video stream by a method comprising passingthe syntax structures of the base layer access units to a base layerbuffer, passing syntax structures of the enhancement layer access unitsto an enhancement layer buffer, outputting the syntax structures passedto the base layer buffer in a predetermined sequence, outputting thesyntax structures passed to the enhancement layer buffer in apredetermined sequence, and recombining the sequences of syntaxstructures output by the base layer buffer and the enhancement layerbuffer respectively to form a complete enhanced access unit, comprisingbase layer syntax structures and enhancement layer syntax structures ina predetermined sequence.

In accordance with a eighth aspect of the disclosed subject matter thereis provided a computer readable medium containing software that, whenexecuted by a computer having an input for receiving a unitary bitstreamcomposed of base layer access units and enhancement layer access units,processes the bitstream by a method comprising separating a base layerprogram stream and an enhancement layer program stream from the unitarybitstream, and inserting a delimiting syntax structure into theenhancement layer program stream.

In accordance with a ninth aspect of the disclosed subject matter thereis provided a computer readable medium containing software that, whenexecuted by a computer having an input for receiving a unitary bitstreamcomposed of base layer access units and enhancement layer access units,wherein each enhancement layer access unit comprises video layer syntaxstructures and non-video layer syntax structures, processes thebitstream by a method comprising separating a base layer program streamand an enhancement layer program stream from the unitary bitstream, andincluding non-video layer syntax structures of the enhancement layeraccess units in the base layer program stream, whereby the enhancedvideo signal comprises a base layer component that includes non-videolayer syntax structures of the enhancement layer access units and anenhancement layer component that includes video layer syntax structuresof the enhancement layer access units.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the invention, and to show how the samemay be carried into effect, reference will now be made, by way ofexample, to the accompanying drawings, in which:

FIG. 1 is a block schematic illustration of the architecture of a firstsystem for supplying compressed video material for presentation,

FIG. 2 illustrates the structure of an SVC access unit with base layerand one enhancement layer,

FIG. 3 is a block schematic illustration of the architecture and usecase for transmission of compressed SVC video material for decode andpresentation,

FIG. 4 is a block schematic illustration of the architecture of a secondsystem for supplying compressed video material for presentation,

FIG. 5 is a block schematic illustration of the architecture of a thirdsystem for supplying compressed video material for presentation,

FIGS. 6A, 6B and 6C illustrate the structure of data units generated atvarious points in the system shown in FIG. 5, and

FIG. 7 is a is a block schematic diagram of a computing machine that maybe used to implement parts of the processes described with reference toFIGS. 4 and 5.

In the several Figures of the drawings, like reference signs are used todesignate like or equivalent elements, functions or components.

DETAILED DESCRIPTION

The SVC encoder 10′ shown in FIG. 4 generates a bitstream that conveysthe base layer and one enhancement layer. The NAL separator 52 dividesthe bitstream into two sub-bitstreams conveying, respectively, the baselayer NAL units and the enhancement layer NAL units and the twosub-bitstreams pass to a base layer video packetizer 14 ₀ and anenhancement layer video packetizer 14 ₁. As described with reference toFIG. 3, the two packetizers create respective video PESs. The transportstream multiplexer 54 assigns different PIDs to the two PESs and outputsa transport stream conveying the two layers. The transport streammultiplexer 54 includes a T-STD buffer conforming to the SVC T-STDmodel. The transport stream is transmitted either as an SPTS directly toa transport stream demultiplexer 56 or is multiplexed with othertransport streams and is supplied as part of an MPTS to the transportstream demultiplexer 56 via a program demultiplexer (not shown).

As is conventional, the transport stream demultiplexer 56 separates thetransport stream packets on the basis of PID and depacketizes thetransport stream packets to re-create the PES packets. In this manner,the transport stream demultiplexer outputs both a base layer PES and anenhancement layer PES, as well as one or more audio PESs. As shown inFIG. 4, the base layer PES and the enhancement layer PES are supplied toa video T-STD 60, which comprises a base layer T-STD buffer 64 and anenhancement layer T-STD buffer 68. The base layer T-STD buffer 64,similarly to the T-STD buffer 40 shown in FIG. 1, includes a transportbuffer Tb0 and a multiplexing buffer Mb0. The multiplexing bufferoutputs an encoded bitstream containing the base layer access units (theAVC non-VCL NAL units and the AVC VCL NAL units) to an elementary streambuffer segment ESb0. The size of the buffer segment ESb0 may not exceedthe size of the elementary stream buffer Eb prescribed in the H.264/AVCstandard.

The enhancement layer T-STD buffer 68 also includes a transport bufferTb1, a multiplexing buffer Mb1 and an elementary stream buffer segmentESb1. Similarly to the multiplexing buffer Mb0, the buffer Mb1 outputsan encoded bitstream containing the enhancement layer access units (theSVC non-VCL NAL units and the SVC VCL NAL units) which when combinedappropriately with base layer access units produces an SVC access unitas defined in Annex G of H.264.

The combined size of the buffer segments ESb0 and ESb1 may not exceedthe size of the elementary stream buffer Eb prescribed in Annex G of theH.264/AVC standard for an SVC decoder that decodes a program having abase layer and one enhancement layer. However, the total permittedbuffer size may be allocated between the buffer segments to optimizeperformance of the decoder, provided that the size of the buffer segmentESb0 does not exceed the size of the elementary stream buffer Ebprescribed in the H.264/AVC standard for an AVC decoder.

It will be understood by those skilled in the art that the NAL units ofan access unit received by the transport stream demultiplexer 56 mightnot be in the order required for decoding the access unit. Theelementary stream buffer segments, which receive the encoded bitstreamsprovided by the multiplexing buffers, ensure that the NAL units of eachaccess unit are output in the proper order for decoding. A reassemblyfunction Re-A receives the AVC and SVC NAL units output by the two T-STDbuffers respectively and combines the NAL units in the proper sequenceto re-create the SVC access unit structure shown in FIG. 2. Thereassembly function supplies the SVC access units to the SVC decoder fordecoding at the proper respective decode times. In this manner, theT-STD 60 is able to output pictures that include the enhancementssupported by SVC.

The buffer management (i.e., the sizes of the transport buffer,multiplexing buffer and the combined elementary stream buffer segmentsas well as the transfer rate between the buffers) is the same as in aconventional MPEG-2 T-STD. Data enters the elementary stream buffersegments at the rate specified for the output of data from themultiplexing buffer in the conventional MPEG-2 T-STD model and afterboth the base layer NAL units and the enhancement layer NAL units of agiven SVC access unit are present in the respective elementary streambuffer segments, they are transferred instantaneously to the reassemblyfunction where they are combined and transferred instantaneously to theSVC decoder. Thus, the elementary stream buffer segments and thereassembly function do not introduce any latency between themultiplexing buffer and the SVC decoder.

The transport stream output by the transport stream multiplexer 54 mayalso be supplied to an AVC T-STD 34′ via a transport streamdemultiplexer 30. The transport stream multiplexer 30 separates the baselayer PES from the transport stream and supplies the base layer PES tothe T-STD 34′. Since the enhancement layer PES is not supplied to theT-STD 34′, the T-STD 34′ is not burdened by having to process NAL unitsthat are not needed to decode the base layer access units.

Referring now to FIG. 5, in another embodiment the bitstream produced bythe SVC encoder 10′ conveys a base layer and two enhancement layers ENH1and ENH2. The NAL separator 52 separates the bitstream into threesub-bitstreams based on the NUT values of the NAL units, but unlike inthe case of FIG. 4 the base layer bitstream includes not only the AVCNAL units but also the SVC non-VCL NAL units (including the NAL unitsthat are compatible with AVC, as described above). The base layerbitstream produced by the NAL separator has the structure shown in FIG.6A. The two enhancement layer sub-bitstreams each contain the SVC VCLNAL units for the respective SVC access units. The NAL separator alsoinserts an SVC delim NAL unit at the beginning of each access unit ofeach enhancement layer sub-bitstream, as shown in FIG. 6B. The SVC delimNAL unit may be very small and has a reserved NUT value, such as 23 or24. This SVC delim NAL unit is in addition to the access unit delimiterhaving NUT equal 9, as mandated by the MPEG-2 systems standard.

The three sub-bitstreams are supplied to the packetizers 14 ₀, 14 ₁, and14 ₂ respectively, which create respective PESs and supply the PESs to atransport stream multiplexer 72. The transport stream multiplexer 72,which includes a buffer conforming to the SVC T-STD model, assignsdifferent PIDs to the three PESs and outputs a transport streamconveying the three layers. It will be appreciated that the base layerPES contains all of the AVC NAL units that are required to decode thebase layer access units.

The transport stream created by the transport stream multiplexer 72 issupplied to a transport stream decoding function 74. The transportstream decoding function includes a transport stream demultiplexer 76which separates the base layer PES and the two enhancement layer PESsbased on PID and supplies them to respective T-STD buffers 80, 81 and82. Each T-STD buffer includes a transport buffer Tb, a multiplexingbuffer Mb and an elementary stream buffer segment ESb. The combined sizeof the buffer segments ESb0, ESb1 and ESb2 may not exceed the size ofthe elementary stream buffer Eb prescribed in Annex G of the H.264/AVCstandard for an SVC decoder that decodes a program having a base layerand two enhancement layers. However, the total permitted buffer size maybe allocated among the buffer segments to optimize performance of thedecoder, provided that the combined size of the buffer segments ESb0 andESb does not exceed the size of the elementary stream buffer Ebprescribed in Annex G of the H.264/AVC standard for an SVC decoder thatdecodes a program having a base layer and one enhancement layer and thesize of the buffer segment ESb0 does not exceed the size of theelementary stream buffer Eb prescribed in the H.264/AVC standard for anAVC decoder.

Each T-STD buffer processes the bitstream that it receives in a similarmanner to that described with reference to FIG. 4, such that the bufferDRB0 outputs access units having the structure shown in FIG. 6A and thebuffers DRB1 and DRB2 each output access units having the structureshown in FIG. 6B. The outputs of the buffer structures are concatenatedby a reassembly function Re-A to create the sequence of NAL units foreach access unit as shown in FIG. 6C. The output of the reassemblyfunction passes to the elementary stream buffer, as described withreference to FIG. 4.

FIG. 5 shows that the transport stream output by the multiplexer 54 maybe supplied to an AVC T-STD 34′ via a transport stream demultiplexer 30,as described with reference to FIG. 4.

FIG. 5 also illustrates the three video PESs being supplied over anerror free medium to a program stream decoding function 90. The baselayer PES may be supplied to a base layer AVC P-STD, including a bufferand a decoder, whereas the base layer PES and the two enhancement layerPESs ENH1 and ENH2 are supplied to a program stream multiplexer, whichincludes a buffer that conforms to the SVC P-STD model. The programstream multiplexer supplies the base layer PES and the enhancement layerPES ENH1 to an SVC decoder 91, which includes a program streamdemultiplexer that removes the PES headers from the base layer PES andthe enhancement layer PES and directs the base layer bitstream to anelementary stream buffer segment ESb0 and the enhancement layer ENH1bitstream to an elementary stream buffer segment ESb1. The outputs ofthe buffer segments ESb0 and ESb1 pass to a reassembly function. Thesizes of the buffer segments ESb0 and ESb are variable in similarfashion to the sizes of the elementary stream buffer segments in thebuffers 80 and 81. The operation of the SVC decoder 91 thereforecorresponds, in the program stream context, to the operation of thebuffers 80 and 81 and the reassembly function in the transport streamcontext.

The program stream multiplexer supplies the base layer PES and the twoenhancement layer PESs ENH1 and ENH2 to an SVC decoder 92, which issimilar to the decoder 91 but is augmented by an elementary streambuffer segment ESb2 corresponding to the elementary stream buffer in theT-STD buffer 82. The program stream decoding function 90 is thereforeable to decode either the base layer alone, or the base layer andenhancement layer ENH1, or the base layer and both enhancement layerENH1 and enhancement layer ENH2.

In both the embodiment shown in FIG. 4 and the embodiment shown in FIG.5, available buffer size, based on the limitations in the H.264/AVCstandard, including Annex G, may be allocated dynamically, on anas-needed basis, to the different layers of the bitstream produced bythe encoder. By allowing dynamic flexible allocation of buffer size tothe different layers in this manner, the decoder may reduce theallocation of buffer size to the base layer and increase the allocationof buffer size to one or more of the enhancement layers based on, forexample, complexity of the access units, to facilitate proper decoding.

The SVC delim NAL unit is easily detected by the transport streamdemultiplexer 76 and facilitates separation of the SVC VCL NAL units.The SVC delim NAL unit is not recognized by the decoder and thereforehas no effect on the decoding of the enhancement layer access units.

As described in connection with FIG. 4, the NAL separator may separateboth the SVC non-VCL NAL units and the SVC VCL NAL units from the SVCaccess unit, so that the base layer PES includes only the AVC NAL unitsof the access unit structure shown in FIG. 2, and the enhancement layerPES contains all the SVC NAL units whereas in the case of FIG. 5 the NALseparator groups the SVC non-VCL NAL units with the AVC NAL units, sothat the base layer PES contains the SVC non-VCL NAL units. Thisapproach results in a slight increase in the bandwidth required totransmit the base layer PES, but it will be appreciated from anexamination of the SVC access unit structure shown in FIG. 2 that byincluding the SVC non-VCL NAL units in the base layer PES, reassembly ofthe SVC access unit can be accomplished by simple concatenation of theNAL units included in the base layer PES and the enhancement layer PESrespectively. Otherwise, parsing of the data would be required in thereassembly buffer, which would add latency and complexity to theprocess. The simplicity of reassembly of the SVC access unit offsets theslight disadvantage of requiring slightly greater bandwidth to transmitthe base layer PES.

Referring to FIG. 6, the NAL separator and packetizers shown in FIG. 4or 5, or the transport stream demultiplexer 56 and video T-STD 60 shownin FIG. 4 or the program stream or transport stream decoding functionshown in FIG. 5, may be implemented using a computer comprising at leastone processor 161, random access memory 162, read only memory 163, I/Odevices 164 (including suitable adaptors for receiving and transmittingbitstreams), a user interface 165, a CD ROM drive 166 and a hard diskdrive 167, configured in a generally conventional architecture. Thecomputer operates in accordance with a program that is stored in acomputer readable medium, such as the hard disk drive 167 or a CD ROM168, and is loaded into the random access memory 162 for execution. Theprogram is composed of instructions such that when the computer receivesa bitstream conveying the particular input signal involved, by way of aninput adaptor, the computer allocates memory to appropriate buffers andutilizes other suitable resources and functions to perform the variousoperations that are described above with reference to FIGS. 4 and 5 andtransmits, by way of an output adaptor, a bitstream conveying theparticular output signal.

Although scalable video streams have been discussed above in connectionwith either one or two enhancement layers, it will be appreciated bythose skilled in the art that Annex GI to H.264/AVC allows up to sevenenhancement layers. It will also be appreciated that although theforegoing description of enhancement layers has been in the context ofscalable video, the other types of enhancement to the AVC base layer arepossible.

It will be appreciated that the invention is not restricted to theparticular embodiment that has been described, and that variations maybe made therein without departing from the scope of the invention asdefined in the appended claims, as interpreted in accordance withprinciples of prevailing law, including the doctrine of equivalents orany other principle that enlarges the enforceable scope of a claimbeyond its literal scope. Unless the context indicates otherwise, areference in a claim to the number of instances of an element, be it areference to one instance or more than one instance, requires at leastthe stated number of instances of the element but is not intended toexclude from the scope of the claim a structure or method having moreinstances of that element than stated. The word “comprise” or aderivative thereof, when used in a claim, is used in a nonexclusivesense that is not intended to exclude the presence of other elements orsteps in a claimed structure or method. The appended claims employ terms(such as syntax structure) that are also used in reference documentspertaining to H.264/AVC, but this is by way of convenience for theskilled reader and is not intended to limit the scope of the claims tomethods, apparatus and computer readable media that are dependent on theparticular video coding described in H.264/AVC.

The invention claimed is:
 1. A method of decoding an enhancedmultiplexed transport stream comprising a succession of enhanced accessunits, each comprising a base layer access unit and at least oneenhancement layer access unit, each enhanced access unit comprising aplurality of syntax structures, each of said base layer access unitsincluding a base layer syntax structure, each of said enhancement layeraccess units including an enhancement layer syntax structure, whereinsaid base layer syntax structure is different than said enhancementlayer syntax structure, said method comprising: (a) passing the baselayer syntax structures of the base layer access units to a base layerbuffer, (b) passing the enhancement layer syntax structures of theenhancement layer access units to an enhancement layer buffer, whereinthe enhancement layer syntax structures of the enhancement layer accessunits comprise enhancement layer video layer syntax structures andenhancement layer non-video layer syntax structures, (c) separating thebase layer syntax structures of the base layer access units from thesuccession of enhanced access units, including separating theenhancement layer non-video layer syntax structures of the enhancementlayer access units from the succession; (d) outputting the base layersyntax structures passed to the base layer buffer in a predeterminedsequence, (e) outputting the enhancement layer syntax structures passedto the enhancement layer buffer in a predetermined sequence, and (f)recombining the sequences of syntax structures output by the base layerbuffer and the enhancement layer buffer respectively to form a completeenhanced access unit, comprising base layer syntax structures andenhancement layer syntax structures in a predetermined sequence;wherein: (g) the base layer syntax structures passed to the base layerbuffer include enhancement layer non-video syntax structures; (h) allthe enhancement layer video layer syntax structures of the enhancementlayer access units pass to the enhancement layer buffer; and (i) none ofthe enhancement layer non-video syntax structures of the enhancementlayer access units pass to the enhancement layer buffer.
 2. A methodaccording to claim 1, wherein the enhancement layer syntax structure ofan enhancement layer access unit in the succession of enhanced accessunits comprise enhancement layer video layer syntax structures,enhancement layer non-video layer syntax structures, and a delimitingsyntax structure marking a boundary of the enhancement layer video layersyntax structures of the enhancement layer access unit, and the step ofseparating the base layer syntax structure of the base layer accessunits from the unitary succession of enhanced access units includespassing syntax structures prior to the delimiting syntax structure tothe base layer buffer and passing syntax structures after the delimitingsyntax structure to the enhancement layer buffer.
 3. A method accordingto claim 1, comprising receiving a multiplexed transport streamconveying, with different respective program identifiers, the base layersyntax structures of the base layer access units and remaining syntaxstructures, demultiplexing the multiplexed transport stream to create atleast two program streams based on the different respective programidentifiers, and passing the two program streams to the base layerbuffer and the enhancement layer buffer respectively.
 4. A methodaccording to claim 1, wherein the base layer buffer includes a baselayer elementary stream buffer segment and the enhancement layer bufferincludes an enhancement layer elementary stream buffer segment, and themethod comprises dynamically adjusting the combined sizes of the baselayer and the enhancement layer elementary stream buffer segments tooptimize decoding of the enhanced multiplexed transport stream whilerestricting the maximum size of the base layer elementary stream buffersegment of the base layer buffer.
 5. A method according to claim 1,comprising passing the complete enhanced access unit to a video decoder.6. A method according to claim 1, wherein the enhancement layer accessunits contain enhancement layer syntax structures of at least first andsecond enhancement layers, and the step of passing remaining syntaxstructures to the enhancement layer buffer includes passing syntaxstructures of the first and second enhancement layers to first andsecond enhancement layer buffers respectively.
 7. A method according toclaim 6, wherein the enhanced multiplexed transport stream comprises asuccession of enhanced access units each comprising a base layer accessunit and at least first and second enhancement layer access units, andthe method comprises separating the syntax structures of the base layeraccess units from the succession of enhanced access units.
 8. A methodaccording to claim 7, wherein the enhancement layer syntax structures ofeach enhancement layer access unit comprise enhancement layer videolayer syntax structures and enhancement layer non-video layer syntaxstructures, and the step of separating the base layer syntax structuresof the base layer access units from the succession of enhanced accessunits includes separating enhancement layer non-video layer syntaxstructures of the first and second enhancement layer access units fromthe succession, whereby the syntax structures passed to the base layerbuffer include enhancement layer non-video syntax structures, andwherein all the video layer syntax structures of the first and secondenhancement layer access units pass to the first and second enhancementlayer buffers respectively.
 9. A method according to claim 7, whereinthe syntax structures of an enhancement layer access unit in thesuccession of enhanced access units comprise enhancement layer videolayer syntax structures, enhancement layer non-video layer syntaxstructures, and a delimiting syntax structure marking a boundary of theenhancement layer video layer syntax structures of the enhancement layeraccess unit, and the step of separating the base layer syntax structuresof the base layer access units from the unitary succession of enhancedaccess units includes passing syntax structures prior to the delimitingsyntax structure to the base layer buffer and passing syntax structuresafter the delimiting syntax structure to the enhancement layer buffer.10. A method according to claim 6, comprising receiving a multiplexedtransport stream conveying the base layer syntax structures of the baselayer access units with a first program identifier, enhancement layersyntax structures of the first enhancement layer with a second programidentifier, and enhancement layer syntax structures of the secondenhancement layer with a third program identifier, demultiplexing thetransport stream to create at least first, second and third programstreams based on the first, second and third program identifiersrespectively, and passing the first, second and third program streams tothe base layer buffer and the first and second enhancement layer bufferrespectively.
 11. A method according to claim 6, wherein the base layerbuffer includes an elementary stream buffer segment and the first andsecond enhancement layer buffers includes a first and second elementarystream buffer segments respectively, and the method comprisesdynamically adjusting the combined sizes of the elementary stream buffersegments to optimize decoding of the enhanced multiplexed transportstream while restricting the maximum size of the elementary streambuffer segment of the base layer buffer.
 12. A method according to claim6, comprising passing the complete enhanced access unit to a videodecoder.